Upgrade RocksDB JNI to 10.10.1.1 while preserving 7.9.2 index compatibility - #4886

Motivation

Upgrade RocksDB JNI from 9.9.3 to 10.10.1.1, the latest version published on Maven Central as checked on 2026-09-10, while preserving the bookie's RocksDB index data compatibility with 7.9.2. BookKeeper 4.17.0 uses 7.9.2; later 4.17.x patches use 7.10.2, so 7.9.2 is the conservative compatibility baseline for Pulsar 4.0.x deployments.

Upstream's latest release is 11.8.1, but its rocksdbjni artifact is not available on Maven Central. This PR deliberately targets the latest consumable JNI release, rather than introducing an unresolvable dependency.

The JNI-only 10.10.1.1 release fixes the Windows native-symbol failure reported in RocksDB #14537. Its Java sources are unchanged from 10.10.1; upstream RocksDB source/documentation links therefore retain the v10.10.1 tag.

Changes

• Upgrade rocksdb.version to 10.10.1.1 and update RocksDB artifact/source references in both binary distribution LICENSE files.
• Retain format_version=5 in all three conf/*_rocksdb.conf.default templates and document the benefits and minimum reader versions of formats 6 and 7.
• Explicitly set track_and_verify_wals=false to preserve WAL recovery by 7.9.2, with comments explaining when the newer integrity checks can be enabled.
• Document compatible HyperClockCache, Ribbon filter, and partitioned-filter optimizations with upstream documentation links. Keep the existing checksum, compression and entry-location tuning.
• Update the corresponding test configurations and add a test that writes/compacts data and checks the actual SST footer version for every database type, with both configuration files and Java defaults.

Backwards compatibility at the storage format level

The default bookie ledger-metadata and entry-location indexes remain readable and writable by RocksDB 7.9.2 after upgrading to 10.10.1.1. No index migration or rewrite is required for this configuration. This concerns RocksDB indexes; the BookKeeper journal and entry-log formats are unchanged.

Storage compatibility does not mean configuration files can be copied unchanged between releases: 7.9.2 rejects the new track_and_verify_wals option even when false. Use the older release's configuration on rollback, retaining compatible format/compression settings; do not load the newer generated OPTIONS-* file into the older strict parser. Operators using custom options must preserve these constraints. Lowering format_version does not convert SSTs already written in a newer format; those need rewriting with a capable RocksDB version before rollback. Enabling newer WAL records likewise cannot be undone merely by changing the option back.

This compatibility statement is scoped to the bookie's KeyValueStorageRocksDB indexes and their supplied/default settings, not the separate stream state-store implementation or arbitrary opt-in RocksDB features.

Release-note review

Reviewed the published release notes since 9.9.3 and cumulative history through 11.8.1, including patch-release notes. Relevant changes:

• 9.10–9.11: WAL tracking is opt-in; manual leveled compactions respect max_compaction_bytes more strictly.
• 10.0–10.2: removed obsolete compression/API options are not used by the supplied configuration; writing SST formats below 2 is unsupported; OPTIONS persistence errors now always fail the operation. Custom configurations must remove deleted options such as fail_if_options_file_error and max_write_buffer_number_to_maintain.
• 10.4–10.6: format 7 is opt-in; file-size validation on open is strengthened; periodic compaction scheduling changes; decoupled partitioned filters become the default without breaking SST compatibility.
• 10.7–10.10: native default cache changes to HyperClockCache, parallel compression improves, the LZ4 performance regression is fixed, MANIFEST auto-tuning is added, and the range-deletion compatibility bug is fixed. Explicit entry-location LRU caches remain explicit. These cache/compaction changes do not require an index-format migration.
• Beyond this PR (10.11–11.8.1): format 7 becomes the upstream default; 11.0 drops reads of SST formats 0/1, which would require pre-upgrade compaction for such legacy files; 11.5 changes default compression to LZ4, still readable by 7.9.2. New custom compression, embedded blobs and other experimental formats are not enabled by this PR. A future 11.x JNI upgrade needs its own validation.

Validation

• Passed 13 focused tests, including all six database-type/configuration combinations in the new SST-format test:
  mvn -B -pl bookkeeper-server,stream/statelib -am test -Dtest=KeyValueStorageRocksDBTest,KeyValueStorageTest,EntryLocationIndexTest,LedgerMetadataIndexTest -Dsurefire.failIfNoSpecifiedTests=false
• Passed mvn -B -pl bookkeeper-server -am checkstyle:check apache-rat:check.
• Passed an additional standalone JNI test for each of the three templates: 7.9.2 → 10.10.1.1 → 7.9.2 → 10.10.1.1, using separate JVMs/native libraries. Wrote 2,000 initial records, upgraded and applied point/range deletions plus 2,000 new records, flushed/compacted, then halted the JVM after an additional synced but unflushed write. The older version successfully recovered that WAL, verified all expected values/deletions, and wrote/compacted more data; the upgraded version then reopened and verified again. Older JVMs used the previous configuration files.
• Full BookKeeper CI passed on b787f29832: all test groups, integration and backward-compatibility tests, Java 17/21 compatibility, Windows/macOS builds, binary license validation, and the aggregate CI check. CodeQL also passed. No flaky-test retries were needed for this commit.